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ABSTRACT 

The  authors  propose  an  approach  for  studying  organizational  effec- 
tiveness which  combines  a  modified  version  of  the  Barnard  and  Simon 
Theory  of  Organizational  Equilibrium  (March  &  Simon,  1968)  with  Yuchtman 
and  Seashore's  (1967)  conceptualization  of  organizational  effectiveness 
as  the  bargaining  position  of  the  organization  with  its  various  con- 
stituents. The  focus  of  the  approach  is  not  on  the  assessment  of  the 
bargaining  position  but  rather  on  the  discovery  of  judgment  models , 
called  policies,  which  will  predict  the  judged  effects  on  the  bargain- 
ing position  of  changes  in  the  organization's  product  attributes.   The 
results  of  an  application  of  this  approach  are  presented  for  illustra- 
tive purposes. 


Despite  the  many  different  definitions  of  organizational  effectiveness, 
there  are  only  two  basic  approaches  to  the  empirical  study  of  this  elusive 
concept.  One  approach,  called  the  goal  approach,  conceptualizes  effective- 
ness as  the  extent  of  accomplishment  of  one  or  more  of  the  focal  organiza- 
tion's goals.   If  these  goals  are  those  stated  by  the  organization's  de- 
cision makers,  the  approach  ie  called  a  prescribed -goal  approach;  if  these 
goals  are  thot'e  inferred  by  the  researcher  to  be  either  the  actual  or  the 
normative  goals  of  the  organization,  the  approach  is  said  to  be  a  derived- 
goal  approach  (cf.,  Etzioni,  1961).  The  other  basic  approach,  which  also 
has  several  variations,  is  the  systems  approach.   Systems-approach  advo- 
~*.tes,  similar  to  goal -approach  advocates',  eventually  identify  or  derive 
a  goal  of  the  focal  organization.  But  researchers  using  the  systems  ap- 
proach usually  take  a  much  broader  perspective.  They  study  several  or  all 
of  the  major  processes  of  the  system  (e.g.,  input,  throughput,  output)  and 
stress  the  effective  allocation  of  resources  to  those  processes  or  activ- 
ities. Gross  (1965),  for  example,  identifies  seven  activities  ranging  from 
the  acquisition  of  resources  to  the  satisfaction  of  various  interests  of 
constituents  which  roust  be  considered  *  hen  assessing  the  effectiveness  of 
an  organization.   Systems  approach  advocates,  by  eventually  defining  a  goal 
for  the  systfr.n,  achieve  an  approach  that  is  quite  compatible  with  the  goal 
approach.  The  essential  difference  between  these  approaches  is  the  breadth 
of  the  perspective  taken  by  the  researcher  and  the  criteria  used  to  assess 
effectiveness. 

A  more  serious  matter  is  the  difference  of  opinion  among  researchers  con- 
cerning the  measurement  of  organizational  effectiveness.  Viewpoints  usually 
fall  within  one  of  the  following  two  broad  categories:   (1)  effectiveness 
should  be  measured  in  terras  of  the  organization's  contribution  to  society, 


or  (2)  effectiveness  should  be  measured  in  terms  of  the  organization's  re- 
ceipts from  society  (Friedlander  &  Pickle,  1968).  Parsons  (1960),  for 
example,  insists  that  organizations  ex^st  to  benefit  society;  therefore, 
society's  needs  should  be  emphasized  when  assessing  organizational  effec- 
tiveness.  Yuchtman  and  Seashore  (1967)  object,  insisting  that  contribu- 
tions to  society  are  costly  requirements  rather  than  indicators  of  success. 
Yuchtman  and  Seashore  (1967)  define  the  effectiveness  of  an  organization  in 
terms  of  "its  bargaining  position,  as  reflected  in  the  ability  of  the  or- 
ganization, in  either  relative  or  absolute  terms,  to  exploit  its  environ-  • 
ment  in  the  acquisition  of  scarce  and  valued  resources"  (p.  55).   They 
stress  that  the  potential  of  the  environment  must  be  considered,  but  that 
r*r».e  must  be  careful  not  to  infer  that  the  organization's  ability  to  exploit 
its  environment  means  the  organization  should  make  maximum  short  term  use 
of  that  ability. 

Yuchtman  and  Seashore's  (1967)  approach  also  suggests  another  important 
difference  among  researchers  in  the  area.  They  emphasize  "the  distinctive- 
ness of  the  organization  as  an  identifiable  social  structure  or  entity" 
(p.  54).   The  conceptualization  of  an  organization  as  an  entity  implies 
that  effectiveness  is  the  bargaining  position  of  the  organization  with 
holders  of  resources  outside  the  organization.   There  are,  of  course,  dif- 
ficulties in  identifying  who  or  what  is  inside  the  organization  and  who  or 
what  is  outside  the  organization.   We  prefer  to  conceptualize  an  organi- 
zation as  a  process  or  a  way.   Such  a  conceptualization  makes  it  unnecessary 
to  determine  whether  a  particular  constituent  is  inside  or  outside  the  or- 
ganization.  It  is  more  appropriate,  in  our  opinion,  to  consider  whether 
a  constituent  (as  a  holder  of  a  scarce  and  valued  resource)  is  a  participant 


in  a  given  process.  A  constituent,  therefore,  may  be  either  inside  or  out- 
side of  trie  organization  U3ing  that  process. 

Another  difference  of  opinion  apparent  among  the  present  approaches  to 
the  study  of  effectiveness  concerns  their  purpose.   The  main  purpose  of 
virtually  all  reported  studies  of  effectiveness  is  to  assess  the  organi- 
zations current  condition.   But  practitioners  usually  face  another  impor- 
tant question-;   How  can  effectiveness  be  increased? 

In  summary,  the  present  differences  of  opinion  in  the  area  of  organiza- 
tional effectiveness  deal  with  choice  of  criteria,  viewpoint,  and  purpose, 

A  possible  resolution.   The  what -criteria  and  whose-viewpoint  questions 
*re  obviously  related  if  one  accepts  the.  Barnard-Simon  Theory  of  Organisa- 
tional Equilibrium  (March  &  Simon,  1958).  This  virtually  tautological 
theory  indicates  that  each  holder  of  a  scarce  resource  will  use  his  own 
criteria  when  judging  the  worth  of  an  inducement  being  offered  in  exchange 
for  a  scarce  resource.   Hence,  there  will  be  many  criteria  and  many  view- 
points and  an  approach  to  the  study  of  effectiveness  should  be  compatible 
with  this  presumed  fact.  Also,  the  theory  indicates  that  the  higher  the 
holder  of  a  scarce  resource  judges  the  worth  of  an  inducement ?  the  greater 
the  amount  of  resources  he  will  be  willing  tc   contribute.   The  Barnard- 
Simon  theory,  therefore,  is  consistent  with  the  concept  of  effectiveness 
as  a  bargaining  position  as  long  as  the  researcher  recognizes  that  there 
are  many  inducements  and  many  constituents,  as  well  as  many  subsequent  bar- 
gaining posit  ions . 

Fricdlander  and  Pickle  (1968),  in  effect,  used  a  modified  and  operation- 
alized  version  of  Yuchtman  and  Seashore1 s  (1967)  concept  of  effectiveness 
and,  to  some  extent,  certain  aspects  of  the  Barnard-Simon  Theory  of  Organ- 
izational Equilibrium  (March  &  Simon,  1958).   Friedlander  and  Pickle  used 
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judgmental  data  to  assess  the  effectiveness  of  an. organization  from  the 
viewpoints  of  seven  different  constituent  groups.   Their  use  of  judgmental 
data  is  compatible  with  the  assertions  of  several  other  esearchers  in  the 
area  (see,  for  example,  Katz  &  Kahn,  1966,  and  Pinto  &  Pinder,  1972).   But, 
like  other  effectiveness  studies,  their  focus  on  assessment  leaves  the 
question  of  how  to  increase  effectiveness  unanswered. 

THE  JUDGMENTAL  APPROACH 

t 

The  judgmental  approach  suggested  here  will  complement  Friedlander  and 
Pickle's  (1968)  approach  by  capturing  and  comparing  the  rating  policies 
used  by  various  holders  of  scarce  resources  to  rate  the  products  of  a  par- 
ticular organizational  process.  A  policy  is  defined  as  a  mathematical 
model  relating  the  quantified  rating  given  a  product  (the  stimuli)  to  the 
quantified  quality  dimensions  describing  that  product.   The  focus  of  the 
judgmental  approach,  then,  is  on  the  judges,  not  on  the  stimuli  being 
judged. 

Previous  studies  in  the  area  of  jut"  jraent  indicate  ths*:  judged  ratings 
may  depend  on  properties  of  the  judge  and/or  properties  of  the  stimuli  being 
judged  (Hunt,  1958;  Klein,  Berkovitz,  &  Malone,  1961).   Hence,  if  a  research- 
er is  only  interested  in  inferring  properties  of  the  stimuli,  he  must  con- 
trol for  the  possibly  differing  properties  of  the  judges.   If,  on  the  other 
hand,  he  is  interested  only  in  inferring  properties  of  the  judges,  such  as 
their  policies  concerning  certain  stimuli,  he  must  control  for  the  possibly 
differing  properties  of  the  stimuli.  To  illustrate,  one  judge  may  rate  a 
certain  automobile  three  on  a  seven-point  scale  because  he  thinks  the  auto- 
mobile gets  twenty  miles  per  gallon  of  gasoline,  an  attribute  which  he 


insiders  Important  in  his  policy  for  rating  automobiles.  Another  judge, 
even  though  he  has  an  identical  policy,  may  rate  the  same  automobile  six 
on  the  same  sca^ e  because  he  thinks  the  automobile  gets  thirty  miles  per 
gallon.   If  the  researcher  wants  to  capture  and  compare  the  policies  of 
these  judges  he  must  control  for  these  perceived  differences  In  the  same 
automobile's  properties.   Requiring  each  judge  to  use  the  same  measured 
results  of  the  relevant  attributes  of  the  stimuli  being  rated  provides 
the  necessary  control.   That  is,  if  all  judges  are  using  the  same  descrip- 
tions of  the  stimuli,  yet  their  ratings  of  the  stimuli  differ,  then  one  can 
assume 'the  judges  differ  regarding  the  importance  of  the  dimensions. 
The  basic  steps  of  the  judgmental  approach  are  outlined  below. 

1.  Identify  the  products  of  an  organization  (i.e.,  the  products 
of  a  selected  organizational  process). 

2.  Identify  the  constituents  (holders  of  scarce  resources)  whose 
resources  are  needed  by  the  process  to  produce  those  produces. 

3.  Sample  these  constituents*  rating  behavior  by  asking  them  to 
each  rate  a  number  of  the  products  of  the  given  process.   Con- 
trol for  possibly  different  perceptions  of  the  products  by 

1 
describing  the  products  similarly  to  each  judge. 

4.  Regress  each  judge's  quantified  ratings  of  the  products  on 
the  quantified  quality  dimensions  describing  the  products. 
These  regression  equations  (one  for  each  judge)  are  called 
policies;  their  R-squares  indicate  the  degree  to  which  a 
judge's  policy  has  been  captured. 

The  beta  weights  in  a  judge's  policy  Indicate  the  relative  weights  which  that 
judge  places  on  each  of  the  quality  dimensions  when  arriving  at  an  overall 
rating  of  product  quality.2  These  policies  allow  the  decision  maker  to 
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assess  the  effects  that  changes  in  a  single  attribute  will  have  on  the 
products*  overaj 1  rating  (or  effectiveness). 

It  is  assumed  that  the  constituents  ^rom  whom  the  judgmental  product 
ratings  are  obtained  will  perceive  a  high  positive  correlation  between  any 
two  of  the  following:   (1)  the  contributions  made  to  the  organization, 

(2)  the  inducements  resulting  from  participation  in  the  organization,  and 

(3)  the  quality  of  products  produced  by  the  organization.   Underlying  this 
assumption  Is   the  assumption  that  at  least  one  mutually  shared  goal  exists 
among  an  organization's  constituents.   The  accomplishment  of  this  super- 
ordinate  goal  facilitates  goal  accomplishment  for  each  of  the  constituents 
vr   constituent  groups.   High  quality  products  appear  to  be  one  such  goal 
in  the  study  reported  in  the  following  section. 

AN  APPLICATION  OF  THE  JUDGMENTAL  APPROACH 

The  nature  of  the  research  problem  in  this  study,  that  of  capturing  and 
comparing  the  rating  policies  of  an  organization's  constituents,  is  such 
that  any  organization  with  a  definable  product  would  have  sufficed.   How- 
ever, our  biases  were  such  that  we  wanted  to  pick  an  organization  which  ±l. 
widely  found  In  this  country  and  which  is  of  contemporary  interest  to,  and 
has  a  relatively  large  impact  on,  a  large  mass  of  people.  A  large  state- 
supported  university  meets  these  criteria, 

O'Neill  (1971)  says  the  raison  de  en.tr e  of  most  academic  institutions 
is  student  instruction.   Etzioni  (1961)  and  Brown  (1970)  agree.   Perrow 
(1970)  asserts  that  "...  colleges  and  universities  are  judged  in  large 
part  by  the  quality  of  their  product"  (p.  129).   For  professional  education, 
Feinstein  et  al.  (1971)  also  recommend  focusing  on  the  output.   Lawrence 


et  al.  (1970)  indicate  that  universities  must  become  aware  cf  how  students^ 
faculty,  private  industry,  and  other  constituents  weight  the  attributes  of 
the  elusive  product,  higher  education,  is  a  first  step  in  assessing  the 
effectiveness  of  an  institution  of  higher  learning.   The  present  study  rep- 
resents such  an  attempt.   It  provides  the  decision  maker  with  the  rating 
policies  of  individual  constituents.   In  addition,  it  yields  policies  for 
certain  naturally  occurring  groups  (recruiters,  faculty  members,  students), 
as  well  as  a  combination  of  these  and  certain  other  groups  formed  on  the 
basis  cf  che  similarities  of  the  members'  rating  policies. 

The  focal  organization  (i.e.,  the  focal  process)  in  this  study  was  the 
process  used  by  the  Department  of  Accountancy  at  the  University  of  Illinois 
at  Urbana -Champaign  to  produce  (educate)  accountancy  students.   Seniors 
graduating  with  majors  in  accountancy  were  said  to  be  products  of  this  pro- 
cess, recognizing,  of  course,  that  there  are  many  other  products.  Three 
groups  of  constituents  were  assumed  to  greatly  influence  the  quality  of 

these  studenLs:   faculty  members,  students,  and  the  so-called  "Big-8"  public. 

3 
accounting  firms.   It  was  assumed  that  these  three  groups  of  holders  of 

scarce  resources  *?ould  perceive  a  high  positive  correlation  between  any  two 

of  the  following:   (I)  the  contributions  made  to  the  organization,  (2)  the 

inducements  resulting  froTn  participation  in  the  orgar/zation,  and  (3)  the 

quality  of  students  produced  by  the  organization. 

Data  Collection  Procedures.   A  samole  of  50  randomly  selected  accountancy 

majors  whose  names  appeared  on  the  tentative  June,  1973  graduation  list 

served  as  subjects.  They  voluntarily  completed  a  questionnaire  which  asked 

'hem  to  (1)  enter  their  composite  grade  point  average  (GPA) ,  (2)  weight  the 

importance  of  11  selected  job  attributes  against  their  perceptions  of  an 

.J^eal  job,  and  (3)  evaluate  their  position  on  two  items  concerning  their 
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college  education,  The  11  selected  job  attributes  included  ten  occupational 
value  scales  from  Rosenberg's  (1957)  Occupational  Values  Questionnaire  and 
one  additional  scale  developed  to  measure  the  importance  of  working  with 
things  rather  than  people  (essentially  a  reversal  of  one  Rosenberg  scale). 
The  two  educational  items,  which  were  designed  to  capture  philosophy  of 
education,  included  a  scale  derived  from  the  Clark-Trow  Typology  (Trow, 
1962)  to  reflect  either  a  "vocational"  or  "collegiate"  orientation  (i.e., 
emphasis  on  academics  versus  extra-curriculur  activities)  and  one  scale 
concerning  the  purpose  of  an  elective  course  (i.e.,  its  purpose  should  be 
to  deepen  one's  education  rather  than  broaden  it). 

Students'  responses  to  the  11  job  attributes  and  the  two  educational 
items  were  then  evaluated  against  the  following  two  criteria:   (i)  Did  the 
attribute  or  item  discriminate  between  students  (i.e.,  have  a  high  degree 
of  variance)?   (2)  Was  the  attribute  or  item  relatively  independent  in  a 
statistical  sense  (i.e.,  not  highly  correlated  with  any  other  attribute  or 
item)?  Based  on  these  criteria,  several  attributes  were  eliminated.   The 
two  educational  items  were  also  eliminated,  because,  although  they  margin- 
ally met  the  above  criteria,  they  had  not  been  empirically  tested  in  prior 
research. 

Finally,  students'  scores  on  the  attributes  that  were  retained  were  com- 
bined with  their  GPAfs  to  form  a  resume  for  each  student,  providing  a  total 
of  50  student  resumes.   These  resumes  were  used  by  the  judges  to  judge  the 
relative  quality  of  each  student.  A  sample  resume  is  shown  in  Table  1. 


Table  1  about  here 


One  recruiter  from  each  of  the  Big-8  public  accounting  firms  was  selected 
as  a  judge  of  the  students'  resumes.   Each  recruiter  selected  was  based  in 
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Chicago  and  had  visited  the  University  of  Illinois  campus  during  the  spring 
semester  of  that  year,   Eight  accountancy  faculty  members  were  randomly 
selected  as  ju'ges  from  a  list  which  e  .eluded:   (1)  the  department  head, 
(2)  those  faculty  who  were  not  on  campus  at  the  time,  and  (3)  those  faculty 
who  had  not  been  members  of  the  department  or  on  campus  during  the  previous 
three  years.   In  addition,  eight:  randomly  selected  accountancy  majors,  whose 
names  appeared  on  the  tentative  graduation  list  for  the  following  semester, 
were  selected  as  student  judges.  The  cooperation  of  all  24  judges  was  ob- 
tained. 

Appointments  were  made  with  each  of  the  24  judges  to  rate  each  of  the 
50  students  based  en  the  five-item  resumes.   Each  judge  was  given  the  fol- 
lowing instructions: 

I  have  50  sheets  of  paper  here  (resumes  shown)  each  of  which  con- 
tains 5  bits  of  information  on  an  actual  student  graduating  from  the 
University  of  Illinois  at  Urbana -Champaign  with  a  major  in  accountancy. 
Pause 
Please  rate  each  student  by  placing  his  resume  in  one  of  seven 
equally-spaced  piles.   This  sorting  aid  will  help  you  (folded  out  a 
sorting  aid  consisting  of  seven  sheets  numbered  1  to  7,  with  labels 
on  piles  1  and  7).   Placing  a  student Ts  resume  in  Pile  No.  7  means 
that  you  rate  this  student  an  "Outstanding  Prospective  Employee  of 
an  Accounting  Firm."  Placing  a  student ?s  resume  in  Pile  No.  1  means 
that  you  rate  this  student  "Not  &   very  good  Prospective  Employee  of 
an  Accounting  Firm,"  Similarly,  Piles  Nos.  2  through  6  allow  you  to 
rate  the  students  between  these  two  extremes.   Please  imagine  that 
the  piles  represent  equally  spaced  points  on  a  yardstick. 
Pau  se 
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After  distributing  all  the  resumes  into  their  appropriate  piles, 
please  rank  those  within  each  pile;  No.  1  for  the  best  prospective 
employee  in  hat  particular  pile;  No   2  for  next  best,  etc.  and  re- 
cord the  Pile.  No.  and  Rank  No.  within  Pile  on  each  resume,   If  you 
will  put  No.  1  on  tops  No.  2  next ,  etc,  in  each  pile,  I  will  record 
the  Pile  Nos.  and  Rank  Nos.  for  you. 
Fause 
I  would  like  to  ask  you  a  few  questions  after  you  have  finished. 
There  is  no  time  limit  and  you  may  shuffle  them  around,  change  your 
mind,  or  use  any  procedure  you  prefer. 

After  the  completion  of  the  above  task,  the  judges  were  psked  through 
direct  questioning  to  distribute  100  points  according  to  the  relative  im- 
portance they  placed  on  each  of  the  five  quality  dimensions.   In  this 
manner,  three  sets  of  judgmental  data  were  collected,  two  in  the  sorting 
of  student  resumes  and  one  through  the  subsequent  direct  questioning. 
The  nature  of  the  latter  set,  subjective  weights  placed  on  student  at- 
tributes, is  straight -forward  and  requires  no  further  explanation.   Some, 
comments  on  the  first  two  sets  of  ratings  seem  warranted. 

Judges  first  rated  each  student  on  a  seven-point  scale  of  attractiveness 
as  a  potential  employee  of  an  accounting  firm.   Then,  by  combining  all  re- 
sumes, the  judges  in  effect  rated  eacn  student  on  a  fifty-point  scale.   In 
addition  to  the  differences  in  the  scale  lengths,  other  important  differences 
appear  to  exist  between  the  two  sets  of  ratings.   The  first  set  of  ratings 
presumably  represented  ratings  according  to  some  preconceived  criterion  (or 
criteria)  used  by  each  judge  for  assessing  the  attractiveness  of  a  potential 
employee  of  an  accounting  firm.   The  second  set  of  ratings  represented  rat- 
ings of  students  relative  to  each  other.   One  might  argue  that  the  subsequent 


ranking  of  the  students  within  each  pile  is  analogous  to  the  use  of  a  ver- 
nier scale  on  the  original  seven-point  sr:ale.   On  the  other  hand,  the  second 
set  of  data  sxay  contain  distortions,  especially  if  the  judges  changed  their 
criterion  (or  criteria)  from  independent  preconceptions  of  attractiveness 
to  dependent  perceptions  relative  to  the  specific  piles  of  students  being 
ranked.  More  research  is  needed  en  the  differences,  If  any.,  resulting  from 
the  two  methods  of  data  collection.   Results  of  the  analyses  of  both  sets 
of  data  are  presented.   No  attempt  is  made  to  interpret  the  differences  in 
the  results  based  on  the  two  sets  of  data, 

ANALYSES  AND  RESULTS 

Using  the  attractiveness  ratings  assigned  the  students  as  dependent  var- 
iables and  the  quality  dimensions  as  independent  variables,  various  multiple 
regression  equations  were  calculated,  one  for  each  judge,  one  for  each  O^oap 
of  judges,  one  for  all  the  judges  combined,  and  so  on.   Researchers  in  the 
area  of  judgment  analysis  call  these  regression  equations  "policies11  or 
"par amorphic  representations"  of  judges'  policies  (Hoffman,  1960),   The  mul- 
tiple R-square;  indicate  the  degree  to  which  a  judge's  policy  has  been  cap- 
tured (Naylor  &  Wherry,  1965).   The  beta  weights  indicate  the  relative 
weights  which  judges  placed  on  each  of  the  quality  dimensions  used  to  des- 
cribe  the  students.   These  beta  weights  are  shown  in  Tables  2  and  3.   The 


Tables  2,  3,  and  4  about  here 


statistical  significance  of  these  beta  weights  are  shown  in  Table  4.   Becaus* 
the  quality  dimensions  were  not  significantly  correlated,  the  beta  weights 
could  be  interpreted  as  relative  weights;  however,  as  the  R-squares  were  not 
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equal  for  each  judge,  comparisons  among  judges  using  the  beta  weights  could 
not  be  accurately  made.   Hoffman's  (1960)  procedure  for  determining  relative 
weights  was  use^  as  it  seemed  that  it  m-fght  be  preferable  for  comparing  the 
weights  placed  on  the  various  quality  dimensions  of  students  across  judges. 
These  relative  weights  are  presented  in  Tables  5  and  6. 


Tables  5  and  6  about  here 


A  visual  inspection  of  Tables  2  through  6  indicates  the  judges  weighted 
the  GPA  dimension  quite  heavily.  Por  all  judges,  an  increase  in  a  student* s 
GPA  resulted  in  a  higher  predicted  rating  of  the  student.  Few  judges  placed 
piuch  relative  weight  on  students*  responses*  to  the  occupational  value  items. 

In  contrast  to  the  visual  method  suggested  above,  the  JAN  hierarchical 

4  i 

grouping  program  is  a  more  sophisticated  procedure  for  comparing  judges 

policies.   The  J*\N  program  starts  with  the  same  number  of  regression  equa- 
tions as  the  number  of  judges.   A  judge's  regression  equation  is  used  to 
predict  that  judge's  ratings.   Since  each  judge's  regression  equation  is 
the  best  equati  n  for  linearly  predict:  .g  that  judge's  ratings,  then  the 
maximum  efficiency  for  predicting  all  1200  ratings  (in  this  case)  would  be 
obtained  by  utilizing  all  24  policies.   The  use  of  fewer  policies,  assuming 
they  are  different  policies,  would  result  in  less  predictive  efficiency. 
The  JAN  program,  therefore,  was  used  to  systematically  reduce  the  number  of 
policies  used  to  predict  all  1200  ratings  by  combining  two  judges'  or  two 
groups  of  judges'  data  in  such  a  manner  that  the  maximum  amount  of  predic- 
tive efficiency  was  retained  (i.e. ,  the  maximum  amount  possible  with  one 
less  policy).   The  drop  in  predictive  efficiency  (R  ),  called  the  drop  in 
composite  group  predictive  efficiency  by  Naylor  and  Wherry  (1.965),  was  tested 
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for  significance  at  each  step.  The  degrees  of  freedom  and  F-values  are 
shown  in  the  last  two  columns  of  Tables  7  and  8.   The  F-value  is  the 


Tables  7  and  8  about  here 


statistic  used  for  testing  the  hypothesis  that  the  two  policies  being 
grouped  at  that  stage  possess  the  same  set  of  beta  weights  (Bottenberg  & 
Christal,  1961;  Naylor  &  Wherry,  1965).   In  the  present  study,  the  degrees 
of  freedom  are  essentially  four  and  infinity  for  all  F-values;  the  rejec- 
tion value  for  p  <  .01  is  3.32.   Hence,  statistically,  there  are  eight 
policies  which  can  be  considered  different  from  each  other  if  Rank  No. 
Data  are  used,  and  seventeen  statistically  different  policies  if  Pile  No. 
Data  are  used. 

Recall  that  the  judges  were  also  asked  to  distribute  100  points  accord- 
ing to  the  relative  importance  they  placed  on  each  of  the  quality  dimensions 
when  rating  the  students.   This  direct  questioning  method  is  the  traditional 
procedure  for  ascertaining  relative  weighting  systems.   Weights  obtained  in 
this  manner  are  sometimes  called  subjective  weights  (Ho  f  Jinan,  1960).   Com- 
parisons of  these  subjective  weights  and  the  empirically  determined  Hoffman's 
(I960)  relative  weights  showed  that  all  judges,  except  No.  16,  grossly  un- 
derestimated the  weight  which  the  empirical  calculations  indicated  they  had 
placed  on  their  major  cue  (the  one  having  the  largest  relative  weight).   For 
most  judges,  the  major  cue  was  GPA.   For  a  more  complete  discussion  of  these 
calculations,  see  Blevins  (1975). 
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SUMMARY  AND  DISCUSSION  OF  RESULTS 

Very  high  R-squares  for  most  of  the  judges*  policies  in  this  study  con- 
firm the  findings  of  other  judgment  studies  in  other  contexts  which  indicate 
that  linear  multiple  regression  equations  will  adequately  describe  a  judge's 
policy  in  a  predictive  sense.  Visual  comparisons  of  these  policies  and  in- 
dexes calculated  from  them  point  out  that  more  than  one  policy  was  being 
used  by  the  judges.  The  relatively  more  sophisticated  JAN  analysis  confirms 
that  more  than  one  policy  was  being  used,  but  the  total  number  of  policies 
was  fewer  than  the  number  of  judges.   The  weights  determined  by  calculations 
from  the  actual  rating  behavior  of  the  judges  differed  from  those  obtained 
by  simply  asking  the  judges  to  state  the  weights  they  used.   In  other  words, 
the  two  methods  yielded  different  results. 

In  terms  of  attempts  to  change  the  bargaining  position  of  the  department 
involved  (i.e.,  increasing  students*  ratings),  the  decision  maker  may  place 
far  too  little  emphasis  on  GPA  and  far  too  much  emphasis  on  occupational 
values  if  he  uses  the  traditional  direct  method  of  questioning  to  determine 
constituents'  rating  policies.   The  assumption  here  is  that  the  empirically 
calculated  weighting  systems  are  more  correct  than  the  directly  solicited 
ones.   From  both  an  intuitive  and  a  statistical  sampling  point  of  view,  the 
calculated  relative  weights  seem  more  valid.   People  frequently  make  judg- 
ments about  other  people;  this  is  the  task  the  judges  were  asked  to  do  in 
this  study.  Rarely,  however,  are  people  required  to  estimate  the  relative 
importance  they  placed  on  each  bit  of  information  they  used  in  judging  other 
people.  From  a  statistical  sampling  point  of  view,  the  rating  behavior  was 
the  focal  phenomenon;  50  observations  of  the  focal  phenomenon  were  made. 
Accordingly,  the  sample  size  of  each  judge's  behavior  was  50.   Using  the 
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alternative  method  in  which  subjective  weights  were  obtained,  the  sample 
size  of  each  judge* s  behavior  was  one;  that  is,  each  judge  was  asked  to 
state  his  weighting  system  only  once.   In  such  a  case,  the  sample  size  of 
the  focal  phenomenon  is  not  increased  by  increasing  the  number  of  judges 
used.   Such  a  procedure  merely  allows  the  researcher  to  make  inferences 
about  some  imaginary  "representative'5  or  "average"  judge.   The  judgmental 
approach  assumes  that  the  decision  maker  wishes  to  make  inferences  about 
real  and  specific  individuals  or  real  and  specific  groups  of  individuals. 
Hence,  multiple  observations  of  each  judge's  behavior  were  made  in  addition 
tc  sampling  across  judges. 

The  judgmental  approach  allows  for  the  fact  that  the  decision  maker  must 
bargain  with  different  constituents  who  hold  different  amounts  and  types 
of  variously  needed  scarce  resources.   These  constituents  will  require  dif- 
ferent amounts  and  types  of  inducements  before  they  are  prepared  to  relin- 
quish those  resources.   Each  constituent  will  assess  the  organization's 
inducements,  using  his  or  her  own  policy. 

The  judgmental  approach  also  requires  that  a  goal  of  the  system  be  iden- 
tified.  In  thi i   study,  that  goal  was  t  le  production  of  students  attractive 
to  accounting  firms.  The  system  goal  was  assumed  to  be  congruent  with  the 
goals  of  students,  faculty,  and  recruiters  who  participated  in  the  process. 
This  approach,  therefore,  is  compatible  with  the  goal  approach  as  well  as 
the  systems  approach.   It  resolves  the  viewpoint  dilemma  since  it  is 
possible,  although  somewhat  impractical,  to  use  the  viewpoint  of  each  and 
every  one  of  the  organization's  constituents.   Furthermore,  the  approach  is 
expected  to  be  directly  useable  by  practitioners.   By  focusing  on  the  effects 
of  changes  In  the  attributes  of  products,  the  results  are  more  directly  re- 
lated to  the  decisions  which  practitioners  must  make.   Regardless  of  the 
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bargaining  position  of  the  organizational  decision  maker,  he  or  she  is  as- 
sumed to  be  interested  in  enhancing  it. 

The  practitioner  using  the  judgmental  approach  may  decide  that  certain 
product  attributes  are  fixed  in  the  sense  that  they  cannot  be  changed  in 
the  production  or  purchasing  process  (recruitment,  in  the  present  example). 
In  such  a  case,  he  would  not  want  to  include  these  attributes  in  the  des- 
cription of  the  products  to  the  judges,  because  they  are  not  relevant  de- 
cision variables. 

Likewise,  a  practitioner  may  decide  that  the  costs  of  catering  to  indi- 
vidual constituents  are  greater  than  the  benefits,  despite  recent  develop- 
ments which  reduce  machine  and  human  information  processing  costs.   In  this 
case,  several  alternatives  are  available  for  assessing  the  effects  of  changes 
in  the  attributes  of  an  organization's  product.  The  decision  maker  may  de- 
cide to  use  only  one  policy  to  predict  the  effects  of  changes  in  product 
attributes.   In  the  present  study,  a  comparable  policy  is  the  last  policy 
listed  in  Tables  2  and  3,  the  one  which  represents  all  24  judges  and  treats 
their  1200  ratings  as  1200  observations  of  one  judge.  Or,  the  decision  maker 
may  decide  to  ut.e  one  policy  for  each  naturally  occurring  group  (i.e.,  faculty 
group,  student  group,  recruiter  group).   These  policies  are  also  shown  in 
Tables  2  and  3.   There  appears  to  be  a  tendency  to  treat  members  of  these 
naturally  occurring  groups  as  if  each  had  the  same  policy.   But  simply  be- 
cause the  individuals  have  similar  titles  does  not,  of  course,  mean  they  will 
exhibit  similar  rating  behavior.   Therefore,  a  decision  maker  may  decide  to 
create  artificial  groups  by  clustering  the  members  accord ing  to  the  criterion 
of  interest;  that  ist  similar  rating  policies.   The  JAN  hierarchical  grouping 
routine  allows  the  decision  maker  to  form  such  groups.   For  example,  Table  8 
indicates  the  decision  maker  can  reduce  the  number  of  policies  used  to  predict 
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all  1200  ratings  from  24  to  9  with  only  a  .01  absolute  drop  in  predictive 
efficiency.  These  nine  policies  are  shown  in  Table  9.  Note  that  five  of 
these  nine  poli-ies  are  individual  judges'  policies.   In  other  words,  even 


Table  9  about  here 


though  the  approach  facilitates  a  solution  to  the  differing  viewpoint  di- 
lemma, it  does  not  require  the  user  of  the  results  to  consider  all  viewpoints 
when  making  decisions  about  product  attributes.   The  decision  maker  may  use 
the  policy  of  each  constituent  or  the  policies  of  any  combination  of  two  or 
more  constituents.  Using  a  fewer  number  of  policies  than  the  total  number 
of  constituents,  results,  of  course,  in  a  loss  in  predictive  efficiency. 
Like  many  managerial  decisions,  there  is  a  tradeoff  between  the  benefits 
of  utilizing  full  information  (each  constituents  policy)  versus  the  cost 
of  getting  and  processing  this  information. 

Several  limitations  of  the  results  of  this  study  should  be  explicitly 
stated.  No  effort  was  made  to  check  the  stability  of  the  policies.   Simi- 
larly, no  attempt  was  made  to  determine  the  function  relating  the  rating 
given  the  product  to  the  amount  of  scarce  resources  the  judge  would  con- 
tribute.  Neither  was  there  an  effort  made  to  ascertain  who  held  how  many 
scarce  resources,  nor  how  much  these  resources  were  valued.   These  data 
must  be  available  before  an  assessment  of  the  overall  bargaining  position 
of  the  organization  can  be  inferred.   Such  an  assessment  was  not  the  focus 
of  this  research.   However,  these  data  might  also  be  useful  to  the  decision 
maker  attempting  to  decide  which  and  how  many  policies  to  use.   For  example, 
a  strong  bargaining  position  with  a  small  holder  of  scarce  resources  is 
probably  not  as  indicative  of  effectiveness  as  is  a  strong  bargaining 
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position  with  a  large  holder  of  scarce  resources.  Regardless  of  which  or 
how  many  policies  the  decision  maker  judges  he  should  use,  the  judgmental 
approach  provides  him  with  the  information  necessary  for  assessing  the 
effects  on  bargaining  position (s)  of  changes  in  the  organization's  product 
attributes,   The  approach  provides  him  with  estimates  of  the  loss  in  pre- 
dictive efficiency  should  he  decide  to  treat  certain  clusters  of  individuals 
as  if  each  member  had  an  identical  policy.   Although  titled  the  judgmental 
approach  because  the  data  are  subjective,  the  researcher  using  this  approach 
retains  as  much  objectivity  as  those  using  approaches  requiring  so-called 
objective  data. 

Finally,  the  reader  will  have  noticed  that  the  system  boundary  in  this 
study  extends  beyond  the  traditional  boundary  of  the  university.   Recruiters 
and  students,  for  example,  are  both  customers  and  producers  of  the  product. 
Recognition  of  this  fact  makes  this  approach  distinctly  different  from  the 
traditional  marketing  approach  to  the  study  of  effectiveness,  which  assumes 
customers  are  outside  and  producers  are  inside  a  given  organization. 
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FOOTNOTES 

1.  If  all  the  products  are  identical  on  each  of  the  quality  dimensions, 
then  product  descriptions  must  be  synthetically  constructed  so  that 
the  measurements  of  the  dimensions  vary  across  products.  A  relative- 
ly high  degree  of  variance  in  the  attributes  is  necessary  because  the 
objective  is  to  discover  the  importance  placed  on  each  dimension.   For 
a  study  using  simulated  stimuli  see  Naylor  and  Wherry  (1965). 

2.  There  is  a  considerable  amount  of  controvery  regarding  the  use  of 
linear  equations  to  represent  a  judge's  policy.   The  reader  who  is 
unfamiliar  with  this  controversy  may  wish  to  read  Hoffman  (1968).   A 
more  recent  discussion  of  this  topic  and  the  use  of  various  relative 
weighting  system  calculations  nay  be  found  in  Dawes  and  Corrigan  (1974). 
A  rather  comprehensive  review  of  judgment  studies  is  contained  in  Slovic 
and  Lichtenstein  (1971). 

3.  Peat  Marwick  Mitchell  &  Co.,  Arthur  Andersen  &  Co.,  Ernst  &  Ernst, 
Coopers  &  Lybrand,  Haskins  &  Sells,  Price  Waterhouse  &  Co.,  Arthur 
Young  &  Co.,  and  Touche  Ross  &  Co. 

4.  The  originating  agency  for  the  hierarchical  grouping  program  used  in 
the  JAN  Model  is  the  Computational  Sciences  Division  of  the  Air  Force 
Human  Resources  Laboratory  at  Lackland  Air  Force  Base,  Texas. 

5.  A  major  criticism  of  judgmental  data  concerns  their  subjectivity;  their 
validity  cannot  be  verified  (Klein,  Berkowitz,  &  Malone,  1961).   These 
criticisms  are  frequently  unwarranted.   To  allow  each  subject  to  use 
his  own  values  or  weighting  system  is  sought  (not  avoided)  in  judgment 
studies.  The  fact  that  the  researcher  is  able  to  reproduce  the  judg- 
ments object ively$  using  the  same  information  as  the  judge,  indicates 
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the  method  is  scientifically  rigorous.   Criticisms  concerning  the  re- 
searcher's subjectivity  while  selecting  the  dimensions  to  be  presented 
to  the  jud£^s  and  the  judges'  possfbly  different  interpretations  of  the 
dimension  descriptions  is  a  more  serious  criticism  (see,  for  example, 
Rommetveit,  1954) .   However,  even  these  criticisms  are  limited  to  judg- 
ment studies  seeking  structural  aspects  of  judges1  policies  and  do  not 
apply  to  studies  using  only  predictive  aspects. 
>♦  Actually,  the  conceptualization  of  an  organization  as  a  process,  as 
opposed  to  an  entity,  does  not  require  the  establishment  of  a  clearly 
defined  boundary. 
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TABLE  1 
SAMPLE  RESUME 


Pile  No. 
Rank  No.  Within  Pile 


Grade  Point  Average   4.320    on  a  5-point  scale 

When  asked: 

"How  heavily  would  you  weight  each  of  the  following  aspects  of  an  Ideal  Job?" 

this  student  gave  the  following  answers: 

PROVIDE  YOU  WITH  ADVENTURE 
Of  No  Importance  ;_ :__ :    :  X  : : Essential 

PERMIT  YOU  TO  BE  CREATIVE  AND  ORIGINAL 
Of  No  Importance  :__ : : :  X  : : Essential 

ENABLE  YOU  TO  LOOK  FORWARD  TO  A  STABLE,  SECURE  FUTURE 
Of  No  Importance  : ___: : : :  X  : Essential 

GIVE  YOU  AN  OPPORTUNITY  TO  WORK  WITH  PEOPLE  RATHER  THAN  THINGS 
Of  No  Importance  :  :  : :    :     :  X   Essential 


TABLE  2 


STANDARDIZED  POLICY  EQUATIONS,  PILE  NO.  DATi 
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Judge 

Number 


GPA 


Beta  Weights  for  Quality  Dimensions 


Adventure 


Creative    Secure    People 


(Decimal  Points  Omitted) 


All  8  Recruiters,  Averaged  Ratings: 

89       01         11 
All  8  Recruiters,  Treated  as  400  Observations; 


9  (Faculty) 
10 
11 
12 
13 
14 
15 
16 


77 

94 
76 
74 
88 
96 
95 
87 
96 


01 

01 
-24 
08 
09 
06 
04 
05 
05 


10 

07 
-35 
28 
02 
01 
00 
05 
-03 


All  8  Faculty,  Averaged  Ratings: 

98        03         02 
All  8  Faculty,  Treated  as  400  Observations: 

84        02         01 


-11 


-10 

21 

-21 

19 

07 

04 

-06 

02 

•03 


04 


03 


18 


16 

00 

-18 

22 

-09 

,07 

06 

22 

06 


06 


05 


R 


1  (Recruiter) 

77 

01 

25 

-17 

05 

80 

2 

95 

05 

00 

03 

08 

95 

3 

88 

04 

11 

00 

19 

94 

4 

i   56 

-20 

05 

-41 

26 

62 

5 

56 

01 

15 

-10 

56 

82 

6 

90 

02 

04 

-17 

07 

91 

7 

81 

02 

14 

-11 

07 

77 

8 

88 

08 

11 

03 

09 

88 

96 


73 

02 
73 
85 
76 
96 
95 
91 
95 


99 


73 


TABLE  2  (continued) 
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Judge 

Number 


Beta  Weights  for  Quality  Dimensicns^ 

GPA    Adventure    Creative    Secure    People 

(Decimal  Points  Omitted) 


All  8  Students,  Averaged  Ratings: 

84  05  12 
All  8  Students,  Treated  as  400  Observations: 

62  09 

All  24  Judges,  Averaged  Ratings: 

92  03  08 
All  24  Judges,  Treated  as  1200  Observations: 

75        02         07 


-21 


-16 


-09 


■07 


22 


16 


15 


i  ? 


R 


17  (Student) 

22 

02 

04 

-72 

15 

61 

18 

49 

-03 

27 

-28 

17 

51 

19 

74 

-06 

14 

02 

42 

91 

20 

81 

09 

20 

18 

24 

93 

21 

90 

05 

15 

04 

11 

95 

22 

;  88 

06 

02 

-05 

27 

96 

23 

58 

02 

09 

-57 

00 

75 

24 

51 

15 

-16 

-04 

11 

32 

94 


53 


H 


64 


29 


TABLE  3 

STANDARDIZED  POLICY  EQUATIONS,  RANK  NO.  DATA 


Judge  Beta  Weights  for  Quality  Dimensions 

Number  GPA    Adventure    Creative    Secure    People    R 


(Decimal  points  omitted) 


1  (Recruiter) 

78 

01 

27 

-09 

11 

83 

2 

93 

02 

03 

-04 

09 

93 

3 

84 

04 

20 

-02 

21 

93 

4 

56 

-19 

05 

-40 

26 

60 

5 

54 

01 

27 

-07 

53 

83 

6 

95 

04 

06 

04 

-03 

91 

7 

84 

-04 

18 

04 

11 

83 

8 

90 

06 

09 

09 

05 

88 

All  8  Recruiters,  Averaged  Ratings: 

88      -01         16        -06       18      95 
All  8  Recruiters,  Treated  as  400  Observations: 

79      -01         14        -06       17      77 


9  (Faculty) 

97 

03 

07 

15 

01 

97 

10 

64 

-42 

-37 

-19 

-13 

71 

11 

76 

03 

26 

24 

21 

88 

12 

88 

06 

01 

13 

-08 

76 

13 

97 

05 

02 

04 

04 

98 

14 

99 

04 

-04 

02 

00 

97 

15 

90 

04 

07 

09 

18 

95 

16 

99 

04 

-03 

04 

00 

98 

All  8  Faculty,  Averaged  Ratings: 

99      -02         00         07        03      98 
All  8  Faculty,  Treated  as  400  Observations: 

89      -02         00         06       03      80 


TABLE  3  (continued) 
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Judge 
Number 


GPA 


Beta  Weights  for  Oualitv  Dimensions 
Adventure 


Creative    Secure    People 


(Decimal  points  omitted) 


17  (Student) 

31 

01 

00 

-73 

18 

68 

18 

63 

03 

23 

-26 

19 

67 

19 

83 

00 

13 

12 

30 

95 

20 

87 

05 

17 

18 

IS 

96 

21 

94 

-01 

17 

10 

06 

98 

22 

89 

06 

02 

-04 

22 

94 

23 

56 

-03 

09 

-65 

-07 

82 

24 

47 

26 

-09 

07 

13 

32 

All  8  Students,  Averaged  Ratings: 

85       06 


11 


All  8  Students,  Treated  as  400  Observations; 

69  05  09 
All  24  Judges,  Averaged  Ratings: 

93  01  09 
All  24  Judges,  Treated  as  1200  Observations 

79       01         08 


-19 


-15 


-06 


-05 


20 


17 


14 


12 


94 


62 


98 


71 


TABLE  4 


SIGNIFICANCE  OF  1ETA  WEIGHTS 
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Bet 

ai 

Bet 

a2 

Beta3 

Beta, 
4 

Beta. 

«4 

Judge 

GPA 

Adven 
Rank 

ture 
Pile 

Creative 
Rank  Pile 

Secure 

Peo 

Rank 

pxe 

Number 

Rank 

Pile 

Rank  Pile 

Pile 

1 

AA 

aa 

A  :     AA 

A 

2 

A* 

aa 

A 

A 

3 

a* 

aa 

AA     AA 

AA 

AA 

4 

AA 

aa 

A 

AA    AA 

A 

A 

5 

** 

a* 

AA      A 

AA 

AA 

6 

AA 

AA 

AA 

7 

AA 

** 

A  ' 

8 

AA 

AA 

A 

9 

AA 

AA 

A  A 

AA    AA 

1C 

AA 

AA 

AA 

AA 

AA     AA 

A      A 

A 

11 

AA 

AA 

AA     AA 

AA    AA 

AA 

AA 

12 

AA 

A* 

13 

AA 

A* 

A 

A 

14 

AA 

AA 

15 

AA 

AA 

A 

AA 

A  A 

16 

AA 

AA 

17 

AA 

A 

AA 

A 

18 

AA 

** 

A      A 

A  A      * 

A 

19 

AA 

AA 

A  A     A  A 

A  A 

•  AA 

AA 

20 

AA 

AA 

A 

AA     AA 

AA     AA 

AA 

AA 

21 

A* 

AA 

AA     AA 

AA 

A 

AA 

22 

AA 

AA 

A 

AA 

AA 

23 

AA 

AA 

AA     AA 

24 

AA 

AA 

* 

**  Significant  at  .01  Level. 
*  Significant  at  .05  Level. 


TABLE  5 

HOFFMAN'S  RELATIVE  WETSHTS  FOR  QUALITY 
DIMENSIONS,  PILE  NO.  DATA 
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Judge 

Number 


GPA     Adventure    Creative    Secure 
(Decimal  points  omitted) 


People 


1  (Recruiter) 

81 

00 

13 

05 

01 

2 

97 

01 

00 

00 

02 

3 

89 

00  . 

03 

00 

07 

4 

58 

02 

01 

26 

13 

5 

48 

00 

06 

01 

45 

6 

93 

00 

•  01  • 

04 

02 

7 

90 

00 

05 

02 

02 

8 

92 

02 

04 

00 

03 

All  8  Recruiters,  Averaged  Ratings: 

88        00 


03 


All  8  Recruiters,  Treated  as  400  Observations: 


88 


00 


03 


All  8  Faculty,  Averaged  Ratings; 

98        00 "        00 
All  8  Faculty,  Treated  as  400  Observations: 

98        00         00 


02 


02 


00 


00 


07 


9  (Faculty) 

95 

00 

02 

03 

00 

10 

69 

08 

13 

07 

03 

11 

70 

01 

14 

03 

11 

12 

99 

02 

00 

00 

-01 

13 

97 

01 

00 

00 

02 

14 

97 

00 

00 

01 

01 

15 

89 

01 

01 

00 

10 

16 

98 

01 

00 

00 

02 

02 


02 


TABLE  5  (continued) 
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Judge 
Number 


GPA     Adventure     Creative 


secure 


(Decimal  points  omitted) 


All  8  Students,  Averaged  Ratings: 

81        01  04 

All  8  Students,  Treated  as  400  Observations: 

81        01  04 

All  24  Judges,  Averaged  Ratings: 

91        00  02 

All  24  Judges,  Treated  as  1200  Observations: 

91        00  02 


06 


06 


01 


01 


People 


17  (Student) 

12 

00 

01 

85 

02 

18 

56 

-01 

20 

1? 

08 

19 

69 

00 

05 

00 

27 

20 

76 

02 

08 

03 

12 

21 

91 

01 

05 

00 

04 

22 

86 

01 

01 

00 

12 

23 

49 

00 

03 

48 

00 

24 

84 

08 

01 

01 

06 

08 


08 


05 


05 


TABLE  6 

HOFFMAN'S  RELATIVE  WEIGHTS  FOR  QUALITY 
DIMENSIONS,    RANI:  NO,    DATA 
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Judge 
Number 


GPA 


Adventure 


Creative 


Secure. 


(Decimal  ooints  omitted) 


All  8  Recruiters,  Averaged  Ratings: 

87        00  06 

All  8  Recruiters,  Treated  as  400  Observations: 


87 


00 


06 


All  8  Faculty,  Averaged  Ratings: 

99        00  00 

All  8  Faculty,  Treated  as  400  Observations: 

99        00  00 


01 


01 


00 


00 


People 


1  (Recruiter) 

81 

00 

14 

02 

04 

2 

96 

00 

01 

01 

03 

3 

82 

01 

08 

00 

09 

4 

59 

02 

02 

25 

13 

5 

44 

00 

14 

00 

41 

6 

99 

01 

.   02 

00 

-01 

7 

89 

00 

07 

00 

04 

8 

95 

01 

03 

00 

02 

07 


07 


9  (Faculty) 

97 

00 

02 

01 

00 

10 

49 

26 

19 

04 

02 

11 

72 

00 

12 

05 

10 

12 

99 

01 

00 

01 

-01 

13 

98 

01 

00 

00 

01 

14 

100 

00 

-01 

00 

00 

15 

90 

00 

02 

00 

07 

16 

100 

00 

00 

00 

00 

01 


01 


TABLE  6  (continued) 
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Judge 

Number 


GPA     Adventure     Creative     Secure 
(Decimal  points  oraitted) 


All  8  Students,  Averaged  Ratings: 

83        01  04 

All  8  Students,  Treated  as  400  Observations: 

83        01  04 

All  24  Judges,  Averaged  Ratings: 

92        00  03 

All  24  Judges,  Treated  as  1200  Observations: 

92        00  03 


05 


05 


01 


01 


People 


17  (Student) 

19 

00 

00 

78 

03 

18 

67 

01 

13 

11 

08 

19 

79 

00 

04 

01 

16 

20 

84 

01 

06 

02 

07 

21 

93 

00 

06 

00 

02 

22 

89 

01 

01 

00 

09 

23 

43 

00 

02 

54 

01 

24 

70 

21 

-01 

01 

09 

08 


08 


05 


05 


TABLE  7 


STEPWISE  AND  CUMULATIVE  LOSS  IN  COMPOSITE  GROUP 
PREDICTIVE  EFFICIENCY  (Rj) ,  PILE  NO.  DATA1 
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Number  of 

cO 

O                 0 

2     's 

Groups 

IT.  -  R  . 

C  1      c  J 

■  -     0  '  ■ 

R  ~     —  IX.  , 

cO    cj 

d.f, 

F 

Remaining 

Stage 

— _ — „_- — — 

— — — 

"" '  ,i- 

24 

0 

,8629 

23 

1 

.8626 

.0003 

.0003 

4/1104 

.6450 

22 

2  T 

.8621 

.0005 

.0008 

4/1108 

1.0235 

21 

3 

.8616 

.0005 

.0014 

4/1112 

1.0832 

20 

4 

.8605 

,0011 

•  .0024 

4/1116 

2.1257 

19 

5 

.8594 

.0011 

.0024 

4/1120 

2.1766 

*   18 

6 

.8583 

.0012 

.0047 

4/1124 

2.3055 

17 

7 

.8566 

.0017 

.0063 

4/1128 

3.3343* 

16 

8 

.8549 

.0017 

.0080 

4/1132 

3.3549* 

15 

9 

.8527 

.0022 

.0102 

4/1136 

4.2647* 

14 

10 

.8504 

.0024 

.0126 

4/1140 

4.6075* 

13 

11 

,8468 

.0036 

.0161 

4/1144 

6.8394* 

12 

12 

.8429 

.0039 

.0200 

4/1148 

7.2847* 

11 

13 

.8280 

.0049 

.0249 

4/1152 

8.9723* 

10 

14 

.8330 

.0050 

.0299 

4/1156 

8.8546* 

9 

15 

.8277 

.0054 

.0353 

4/1160 

9.3316* 

8 

16 

.8209 

.0068 

.0421 

4/1164 

.11.5021* 

7 

17 

.8126 

.0083 

0504 

4/1168 

13.5131* 

6 

18 

.8027 

.0098 

.0602 

4/1172 

15,3942* 

5 

19 

.7880 

.0147 

,0750 

4/1176 

21.9770* 

4 

20 

.7670 

.0210 

,0958 

4/1180 

29.1859* 

3 

21 

.7458 

,0212 

.1171 

4/1184 

26.9490* 

2 

22 

.7026 

.0432 

.1604 

4/1188 

50.5057* 

1 

23 

.6493 

.  0533 

.2136 

4/1.192 

53.3545* 

Format  adapted  from  Naylor  and  Wherry  (1965). 


<  ,01 
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TABLE  8 

STEPWISE  AND  CUMULATIVE  TOSS  IN  COMPOSITE  CROUP 
PREDICTIVE  EFFICIENCY  (R2),  RANK  NO.  DATA1 


Number  of 

*2n 
cO 

2     2 
R  .  -  R  . 

Cl      CJ 

R2n  -  R2 . 
cO    cj 

Groups 

d.f. 

F 

Remaining 

Stage 

24 

0 

,  I  '  3  3 

. 

23 

1 

.8433 

0 

0 

4/1104 

.0264 

22 

2 

.8432 

.0001 

.0002 

4/1108 

.2437 

21 

3 

.8430 

.0002 

.0004 

4/1112 

.3808 

20 

4 

.8427 

.0003 

.0006 

4/1116 

.4695 

19 

5 

.8424 

.0003 

.0009 

4/1120 

.484.8 

18 

6 

.8421 

.0003 

.  0013 

4/1124 

.6167 

17 

7 

.8417 

.  0004 

.0016 

4/1128 

.6524 

16 

8 

.4813 

.0004 

.0020 

4/1132 

.7354 

15 

9 

.8408 

.0005 

.0025 

4/1136 

.8821 

14 

10 

.8401 

.0007 

.0032 

4/1140 

1.1971 

13 

11 

.8393 

.0008 

.0040 

4/1144 

1.4965 

12 

L2 

.8382 

.0011 

.0051 

4/1148 

1.9699 

11 

13 

.8367 

.0015 

.0066 

4/1152 

2.6853 

10 

14 

.8351 

.0015 

.0082 

4/1156 

2.7408 

9 

15 

.8333 

.0019 

.0100 

4/1160 

3.2681 

8 

16 

.8311 

.0022 

.0122 

4/1164 

3.7783* 

7 

17 

.8270 

.0041 

.0163 

4/1168 

7.0882* 

6 

18 

.8211 

.0059 

.0222 

4/1172 

9.9584* 

5 

19 

.8135 

.0077 

.0299 

4/1176 

12.6064* 

4 

20 

.8050 

.0085 

.0384 

4/1180 

13.4412* 

3 

21 

.7862 

.0188 

.0571 

4/1184 

28.5270* 

2 

22 

.7620 

.0242 

.0814 

4/1188 

33.6638* 

1 

23 

.7093 

.0526 

.1340 

4/1192 

65.8532* 

Format  adapted  from  Naylor  and  Wherry  (1965). 
*  p  <  .01 
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table  9 


THE  NINE  POLICIES  REMAINING  AT  STAGE  15,  RANK  NO.  DATA 


Beta  Weights  for  Policies 


Policy   __ 

Number   GPA  Adventure  Creative  Secure  People  Representing  Judge (s) 

(Decimal  points  omitted) 


1 

-56 

19 

-05 

2 

-63 

-03 

-23 

3 

-54 

;   -01 

-2? 

4 

-47 

-26 

09 

5 

-64 

42 

37 

6 

-44 

01 

-05 

7 

-86 

-02 

-14 

8 

-84 

-03 

-16 

9 

-95 

-04 

-04 

40 


26 


07 


■07 


19 


-26 


-19 


-53 


-13 


13 


18 


24 


10 


69  -13  17,23 

03  -15  3,7,1,2,22 

-16  -22  15,20,19,11 

-08  -01  14,16,13,6,12,8,9,21 


